Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Meta-learning, or "learning to learn", refers techniques that infer an inductive bias from data corresponding multiple related tasks with the goal of improving sample efficiency for new, previously unobserved, tasks. A key performance measure meta-learning is meta-generalization gap, is, difference between average loss measured on meta-training and a randomly selected task. This paper presents novel information-theoretic upper bounds gap. Two broad classes algorithms are considered uses either separate within-task training test sets, like MAML, joint Reptile. Extending existing work conventional learning, bound gap derived former class depends mutual information (MI) output algorithm its input data. For latter, includes additional MI per-task learning procedure set capture uncertainty. Tighter then developed, under given technical conditions, two via Individual Task (ITMI) bounds. Applications finally discussed, including noisy iterative meta-learning.